Training Logistic Regression and SVM on 200GB Data Using b-Bit Minwise Hashing and Comparisons with Vowpal Wabbit (VW)

Our recent work on large-scale learning using b-bit minwise hashing [21, 22] was tested on the webspam dataset (about 24 GB in LibSVM format), which may be way too small compared to real datasets used in industry. Since we could not access the proprietary dataset used in [31] for testing the Vowpal Wabbit (VW) hashing algorithm, in this paper we present an experimental study based on the expanded rcv1 dataset (about 200 GB in LibSVM format). In our earlier report [22], the experiments demonstrated that, with merely 200 hashed values per data point, b-bit minwise hashing can achieve similar test accuracies as VW with 10 hashed values per data point, on the webspam dataset. In this paper, our new experiments on the (expanded) rcv1 dataset clearly agree with our earlier observation that b-bit minwise hashing algorithm is substantially more accurate than VW hashing algorithm at the same storage. For example, with 2 (16384) hashed values per data point, VW achieves similar test accuracies as b-bit hashing with merely 30 hashed values per data point. This is of course not surprising as the report [22] has already demonstrated that the variance of the VW algorithm can be order of magnitude(s) larger than the variance of b-bit minwise hashing. It was shown in [22] that VW has the same variance as random projections.